On an extra point, you can only lose the down. Yardage is not accessed on the kickoff. Loss of down means that the team cannot replay that down, therefore, even if the offense scores, the penalty takes the points away.